feat(standard): give each unit a spending cap of its own - #49

Closes #47

What

A run can be given a spending cap per unit beside the run's: nullius standard run --max-usd-per-unit USD, and "Spending cap per problem (USD)" on the builder's Limits step. It is kept in the run's ledger: what each unit spent and has in flight, beside the run's. A call that does not fit its unit's cap is refused by the relay the same way (HTTP 402, recorded refused: unit), and so is every later call of that unit; the run's ledger stays open and the other units run. As decided on the issue:

• a unit's own calls and its simulated user's count toward its cap: they are the task;
• its judge's do not: the judge scores it;
• a unit that reaches its cap ends as UnitSpendingCapError, an Agent error counted against the agent, as one that spends its time is. It had its attempt. A unit the run's cap cut off stays Spending cap, outside the rate.

The benchmark's own relay (an option's embeddings) books for no unit. result.json's budget carries unit_max_usd and over_unit; the job page shows the cap per unit and the units that reached it; merge keeps both. From the builder every job is given the cap per unit whole, where the run's is shared. The parse's cache schema moves to 15.

One change to the run's cap comes with it. A call in flight was booked at the largest reply the run had had from its model. The first real run with a cap per unit showed the flaw: one long reply raised the booking of every later call, and eleven of 24 tasks were cut off at a 0.0009 USD cap having spent as little as 0.0001. A call in flight is now booked at the average reply the run has had from that model (the ledger keeps their count and sum), still 4,096 tokens before any reply and never more than the limit the call names or the model's own.

A review of the three cap pull requests together (#46, #48 and this one) found ten defects; they are fixed here, in 856e6f4, so the three are best merged together:

• a call refused for want of a price was recorded as the cap's and stopped the run: it is recorded apart and the run goes on;
• every POST was booked, and one with no usage (a token count, a cancellation) kept its booking for good: only a call that names its model on a billed route is booked;
• a cap refusal on the benchmark's own relay was not attributed to the unit it cut off;
• the Limits step asked the price of fewer models than the run calls (side models, a control's simulated user);
• the live board said the agents' cost "of the cap", which also counts the judge's calls: the cap is said on its own;
• a run with a cap per unit alone could be recorded stopped and fail formatting a cap it never had;
• the cap's progress line showed as a stage that never starts: the units' stage says why it ended;
• a merged job said the launch stopped at the summed cap when one shard had: it says how many shards stopped at their share;
• the ledger tolerated parts it always writes, was read three times a call, and kept every unit's count: a booking is made under one lock and a unit's count goes when it ends.

A second review found ten more, fixed in 78d0932. The main one: a cap closed for good on estimates alone, so fifteen first calls booked before any reply could end a run nothing had been spent of. A call that would fit with nothing in flight now waits for the calls in flight to settle, two minutes at most; only one that does not fit, or waited in vain, closes the ledger. Also: a call that timed out after it was sent keeps its booking; a ledger that fails is recorded apart, not as the cap; a call that names a model is booked on any route but a token count, and nothing goes out once the cap is reached; a failed driver is the cap's only when the cap refused the driver's own call; a cap per unit under a millionth of a dollar is refused at Limits; a control is not asked the price of a simulated user it never calls, and the run and the Limits step read the models to price from one place (ledger.models); the price list is read at most every five seconds while a recipe is edited. One finding predates the cap and is recorded as #50: a call's estimated cost prices tokens written to a cache at the cache-read rate.

The cap's documentation is also cut to the size of the rows around it (28383e4): the run page's section from 673 words to 257, its longest row from 585 characters to 140.

Stacked on #48.

Touches

backend · routes · templates and static · docs and site · the standard

Before and after

Taken at 1400 px in the dark theme; the images are not attached here.

Verified

• test_standard_ledger.py: a unit that reaches its own cap is refused from then on and the run goes on; a call that is no unit's has none; the run's cap and a unit's are held together and only the run's stops the run; a unit's calls are its own and its simulated user's, never its judge's; the ledger keeps the average reply per model, eight short answers and a long one giving 534 tokens, not 4,000.
• test_standard_relay.py: a unit's own cap refuses its calls and its simulated user's (402, unit, unit_spending_cap), not its judge's nor another unit's, and the benchmark's own relay books for no unit; the refused unit reads UnitSpendingCapError, an Agent error outside the infrastructure outcomes.
• test_standard_run.py: eight units, one of them with a prompt of a thousand tokens over its cap of 0.0005: it alone is cut off, against its agent, the seven others resolve, the run is not stopped, and the result and the live feed say so.
• test_classify.py, test_standard_shard.py, test_standard_recipe.py, app/ui_tests/test_capped_job.py: the outcome on both sides; a merged job keeps a cap per unit and the units that reached it; a recipe's cap per unit goes whole to every job, asks for prices alone, and names Limits when it is no amount; the job page of a run with a cap per unit; the builder keeps it and offers it once.
• Backend suite (1683 tests) and route suite (362 tests), OK, with a test for each defect above but the benchmark relay's attribution and the timeout after a call was sent.
• After the second round, on the real provider: GSM8K, 50 problems 15 at a time under 0.006 USD, which stopped at the start before, sent 31 calls and stopped with 0.0059 spent; HumanEval, 8 tasks 4 at a time under 0.003 USD and 0.0009 a task, sent 26 calls and stopped with 0.0029 spent, where it had stopped with 0.0003. docs-drift script: 0 findings.
• By hand with a real provider (OpenRouter, a model at 0.10 and 0.20 USD a million tokens), every run under a small cap of the run's:
  • HumanEval with the reference agent, 24 tasks 4 at a time, four or five calls a task, --max-usd-per-unit 0.0009 --max-usd 0.05: 21 passed and 3 reached their cap, having spent 0.00073, 0.00076 and 0.00108 (a reply longer than was booked); the run was not stopped and spent 0.0118 of its 0.05. The same run on the largest-reply booking cut 11 tasks off.
  • The same 24 tasks under both caps tight, --max-usd-per-unit 0.0009 --max-usd 0.008: one task reached its own cap and went on record as an Agent error; later the run's cap stopped the run, 3 tasks cut off as Spending cap and 5 not run; 15 passed. The job page shows the notice, both caps and the three outcomes. The run spent 0.0087574 USD, 0.00076 over its cap of 0.008: replies in flight longer than the average they were booked at. Billed 0.0087574.
  • GSM8K from the builder with both caps (0.30 and 0.001 per problem), 10 problems: Review said "Spending cap $0.30" and "Spending cap per problem $0.001"; all ran, and the job page shows both.
  • Four runs' ledgers add up to 0.0259987 USD and the provider billed 0.0259987.
• Browser: the job page's run totals in the dark and light themes; the label column now grows to its longest label, which "Cap per problem" overran.
• Not verified: a benchmark with a simulated user or a judge on a real provider (their routing is tested with stand-ins), the docker backend with a cap per unit, Docker Desktop, a real remote machine.

Risk

• A cap per unit below one call's booking cuts every unit off: before any reply a call is booked 4,096 output tokens (0.0009 USD on this model), so the first units' first calls must fit that. The run's cap no longer has this limit: calls wait for one another.
• A waiting call holds its agent's request open, for two minutes at most. An agent whose own timeout is shorter gives up first.
• Booking at the average, a reply longer than the average costs more than was booked: a unit can pass its cap by that difference (0.00018 USD in the run above), and the run's cap by it for each call in flight (0.00076 USD over a cap of 0.008 with four tasks at a time). The amount depends on the calls in flight, not on the size of the cap.
• A unit over its cap now counts against the agent. A cap set too low for the task reads as the agent failing; the recorded type UnitSpendingCapError says which it was.
• Every cached job is parsed once more after the schema moves.

Undone by a revert: a recipe saved with max_usd_per_unit then reads as one without, and a job's budget keeps its extra keys, unread.